Skip to content

Conversation

jacobbednarz
Copy link

While developing some AMI provisioning tools locally, we attempt to
install the CodeDeploy agent. Unfortunately, the process was hanging
indefinitely.

Digging into the process using strace, it is stuck on a connect to
the EC2 metadata service.

connect(7, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16

This is due to the metadata service not being present in our local
development environment and the installer isn't able to resolve the
service.

After locating this in the installation script, I can see there is a
(very high) read timeout however there isn't an open timeout which is
what the process will use to restrict the amount of time a connection
can be pending a connection. This is particularly useful if the service
you are attempting to connect to is either unresponsive to accept the
connection or not reachable due to things like DNS resolution.

To address the issue, I've added an open timeout to the open call and
now the process fails as expected after a sensible 3s.

Included in this PR is a change extracting out the HTTP options that are
shared by the OpenURI read/open calls to consolidate the
configuration. This also introduces an open_timeout to these
settings to mitigate the potential issue for these calls in the event
the service is unreachable for whatever reason.

While developing some AMI provisioning tools locally, we attempt to
install the CodeDeploy agent. Unfortunately, the process was hanging
indefinitely.

Digging into the process using `strace`, it is stuck on a `connect` to
the EC2 metadata service.

```
connect(7, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16
```

This is due to the metadata service not being present in our local
development environment and the installer isn't able to resolve the
service.

After locating this in the installation script, I can see there is a
(very high) read timeout however there isn't an open timeout which is
what the process will use to restrict the amount of time a connection
can be pending a connection. This is particularly useful if the service
you are attempting to connect to is either unresponsive to accept the
connection or not reachable due to things like DNS resolution.

To address the issue, I've added an open timeout to the `open` call and
now the process fails as expected after a sensible 3s.
Consolidate the HTTP options for OpenURI `read` calls to use the same
settings.

This extends the work done in e7b4ff0 to include a `open_timeout` so
that these calls are not subject to potentially the same issue with
being unable to establish a connection remotely.
@coveralls
Copy link

Coverage Status

Coverage remained the same at 92.007% when pulling d068ba1 on jacobbednarz:define-open-timeout-for-ec2-service-lookups into 81ffec2 on aws:master.

@rohkat-aws
Copy link
Contributor

@jacobbednarz can you use pull request template ?

@jacobbednarz
Copy link
Author

Hey @rohkat-aws!

Thanks for the speedy response! I'm happy to amend my Pull Request to include the mention of the agreement to the Apache 2.0 License however I'm curious regarding the background on why this is a requirement considering the license itself already outlines this is a prerequisite.

From section 5 of the Apache 2.0 license:

  1. Submission of Contributions. Unless You explicitly state otherwise,
    any Contribution intentionally submitted for inclusion in the Work
    by You to the Licensor shall be under the terms and conditions of
    this License, without any additional terms or conditions.
    Notwithstanding the above, nothing herein shall supersede or modify
    the terms of any separate license agreement you may have executed
    with Licensor regarding such Contributions.

Based on my understanding of this, the agreement already assumed and the only reason I would need to mention anything in this space was if I had a separate licensing agreement already in place.

@rohkat-aws
Copy link
Contributor

@hyandell can you please respond to this ?

@jacobbednarz jacobbednarz deleted the define-open-timeout-for-ec2-service-lookups branch March 4, 2019 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants